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Abstract 

In this paper we discuss generic properties of "random subgroups" of 
a given group G. It turns out that in many groups G (even in most ex- 
otic of them) tlie random subgroups have a simple algebraic structure and 
they "sit" inside G in a very particular way. This gives a strong mathe- 
matical foundation for cryptanalysis of several group-based cryptosystems 
and indicates on how to chose "strong keys". To illustrate our technique 
we analyze the Anshel-Anshel-Goldfeld (AAG) cryptosystem and give a 
mathematical explanation of recent success of some heuristic length-based 
attacks on it. Furthermore, we design and analyze a new type of attacks, 
which we term the quotient attacks. Mathematical methods we develop 
here also indicate how one can try to choose "parameters" in AAG to foil 
the attacks. 
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1 Introduction 

Most of the modern cryptosystems use algebraic structures as their platforms 
such as rings, groups, lattices, etc. Typically, cryptographic protocols involve a 
random choice of various algebraic objects related to the platforms: elements, 
subgroups, or homomorphisms. One of the key points to use randomness is to 
foil various statistical attacks, or attacks which could use some specific proper- 
ties of objects if they are not chosen randomly. The main goal of this paper is 
to show that randomly chosen objects quite often have very particular proper- 
ties, which allow some "unexpected" attacks. We argue that knowledge of basic 
properties of the random objects must be a part of any serious cryptanalysis 
and it has to be one of the principal tools in choosing good keys. 

In the paper [37] we studied asymptotic properties of words representing the 
trivial element in a given finitely presented group G. It turned out that a ran- 
domly chosen trivial word in G has a "hyperbolic" van Kampen diagram, even 
if the group G itself is not hyperbolic. This allows one to design a correct (no 
errors) search decision algorithm which gives the answer in polynomial time on 
a generic subset (i.e., on "most" elements) of the Word Search Problem in G. 
A similar result for the Conjugacy Search Problem in finitely presented groups 
has been proven in |38| . These results show that the group-based cryptosystems 
whose security is based on the word or conjugacy search problems are subject 
to effective attacks, unless the keys are chosen in the complements of the cor- 
responding generic sets. Rigorous proofs of results of [37] and [38] are available 
in [SD]. 

In this paper we study asymptotic properties of finitely generated subgroup 
of groups. We start by introducing a methodology to deal with asymptotic 
properties of subgroups in a given finitely generated group, then we describe 
two such properties, and finally we show how one can use them in cryptanalysis 
of group based cryptosystems. 

Then we dwell on the role of asymptotically dominant properties of sub- 
groups in modern cryptanalysis. We mostly focus on one particular example - 
the AAG cryptosystem [5], however, it seems plausible that a similar analysis 
applies to some other cryptosystems. One of our main goals here is to give 
mathematical reasons why the so-called Length Based Attacks give surprisingly 
good results in breaking AAG. Another goal is to introduce and analyze a new 
attack that we coiled a quotient attack. We also want to emphasize that we 
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believe that this " asymptotic cryptanalysis" provides a good method to choose 
strong keys (groups, subgroups, and elements) for AAG scheme (with different 
groups as the platforms) that may prevent some of the known attacks, including 
the ones discussed here. 

The main focus is on security of the Anshel-Anshel-Goldfeld (AAG) public 
key exchange scheme [5] and cryptanalysis of the Length Based Attack (LB A). 
This attack appeared first in the paper [26] by J. Hughes and A. Tannenbaum, 
and later was further developed in a joint paper [23j by Garber, Kaplan, Teicher, 
Tsaban, and Vishne. Recently, the most successful variation of this attack for 
braid groups was developed in [5S]. Notice that Ruinsky, Shamir, and Tsaban 
used LBA in attacking some other algorithmic problems in groups |45] . Our goal 
is to give mathematical reasons why Length Based Attacks, which are, in their 
basic forms, very simple algorithms, give surprisingly good results in breaking 
AAG scheme. It seems plausible that a similar analysis applies to some other 
cryptosystems. We hope that this cryptanalysis provides also a good method 
to choose strong keys (groups, subgroups, and elements) for various realizations 
of AAG schemes that would prevent some of the known attacks. 

The basic idea of LBA is very simple, one solves the Simultaneous Conjugacy 
Search Problem relative to a subgroup (SCSP*) (with a constraint that the so- 
lutions are in a given subgroup) precisely in the same way as this would be done 
in a free group. Astonishingly, experiments show that this strategy works well 
in groups which are far from being free, for instance, in braid groups. We claim 
that the primary reason for such phenomenon is that asymptotically finitely 
generated subgroups in many groups are free. Namely, in many groups a ran- 
domly chosen tuple of elements with overwhelming probability freely generates 
a free subgroup (groups with Free Basis Property). This allows one to ana- 
lyze the generic complexity of LBA, SCSP*, and some other related algorithmic 
problems. Moreover, we argue that LBA implicitly relies on fast computing of 
the geodesic length of elements in finitely generated subgroups of the platform 
group G, or some good approximations of that length. In fact, most of LBA 
strategies tacitly assume that the geodesic length of elements in G is a good 
approximation of the geodesic length of the same elements in a subgroup. On 
the first glance this is a provably wrong assumption, it is known that even in 
a braid group -B„, n > 3, there are infinitely many subgroups whose distortion 
function (that measures the geodesic length in a subgroup relative to the one 
in G) is not bounded by any recursive function. We show, nevertheless, that, 
again, in many groups the distortion of randomly chosen finitely generated sub- 
groups is at most linear. Our prime objective is the braid group Bn,n > 3. 
Unfortunately, the scope of this paper does not allow a thorough investigation 
of asymptotic properties of subgroups of _B„. However, we prove the main re- 
sults for the pure braid groups PB^, which are subgroups of finite index in the 
ambient braid groups. We conjecture that the results hold in the groups i?„ as 
well, and hope to fill in this gap elsewhere in the future. In fact, our results 
hold for all finitely generated groups G that have non-abelian free quotients. 

While studying the length based attacks we realized that there exists a new 
powerful type of attacks on AAG cryptosystems - the quotient attacks (QA). 
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These attacks are just fast generic algorithms to solve various search problems 
in groups, such as the Membership Search Problem (MSP) and SCSP*. The 
main idea behind QA is that to solve a computational problem in a group G 
it sufRces, on most inputs, to solve it in a suitable quotient G/N, provided 
G/N has a fast decision algorithm for the problem. Robustness of such an 
algorithm relies on the following property of the quotient G/N: a randomly 
chosen finitely generated subgroup of G has trivial intersection with the kernel 
A^. In particular, this is the case, if G/N is a free non-abehan group. Notice, 
that a similar idea was already exploited in [55] , but there the answer was given 
only for inputs in "No" part of a given decision problem, which, obviously, does 
not apply to search problems at all. The strength of our approach comes from 
the extra requirement that G/N has the free basis property. 

More generally, our main goal concerns with the methods on how to use 
asymptotic algebra and generic case complexity in cryptanalysis of group based 
cryptosystems. All asymptotic results on subgroups, that are used here, are 
based on the notion of an asymptotic density with respect to the standard dis- 
tributions on generating sets of the subgroups. Essentially, this notion appeared 
first in the form of zero-one laws in probability theory and combinatorics. It 
became extremely popular after seminal works of Erdos, that shaped up the so- 
called The Probabilistic Method (see, for example, [T]). In infinite group theory 
it is due mostly to the famous Gromov's result on hyperbolicity of random 
finitely generated groups (see [32] for a complete proof). Generic complexity 
of algorithmic problems appeared first in the papers [29l [SO] [TOl [9] . We refer 
the reader to a comprehensive survey [24j on generic complexity of algorithms. 
Some recent relevant results on generic complexity of search problems in groups 
(which are of the main interest in cryptography) can be found in |50j . 

This paper is intended to both algebraists and cryptographers. We believe, 
that AAG cryptosystem, despite being heavily battered by several attacks, is 
very much alive still. It simply did not get a fair chance to survive because of 
insufficient group theoretic research it required. It is still quite plausible that 
there are platform groups G and methods to chose strong keys for AAG which 
would foil all known attacks. To find such a group G is an interesting algebraic 
problem. On the other hand, our method of analyzing generic complexity of 
computational security assumptions of AAG, which is based on the asymptotic 
behavior of subgroups in a given group, creates a bridge between asymptotic 
algebra and cryptanalysis. This could be applicable to other cryptosystems 
which rely on a random choice of algebraic objects: subgroups, elements, or 
homomorphisms . 

2 Asymptotically dominant properties 

In this section we develop some tools to study asymptotic properties of sub- 
groups of groups. Throughout this section by G we denote a group with a finite 
generating set X. 
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2.1 A brief description 

Asymptotic properties of subgroups, a priori, depend on a given probability 
distribution on these subgroups. In general, there are several natural procedures 
to generate subgroups in a given group. However, there is no a unique universal 
distribution of this kind. We refer to [3S] for a discussion on different approaches 
to random subgroup generation. 

Our basic principle here is that in applications one has to consider the partic- 
ular distribution that comes from a particular random generator of subgroups 
used in the given application, say a cryptographic protocol. As soon as the 
distribution is fixed one can approach asymptotic properties of subgroups via 
asymptotic densities with respect to a fixed stratification of the set of subgroups 
(which usually comes alone with the generating procedure). We briefly discuss 
these ideas below and refer to [21 [IHl [Ml [SO] ; and to a recent survey [H] , for a 
thorough exposition. In Section [^?^ we adjust these general ideas to a particular 
way to generate subgroups which is used in cryptography. 

Recall, that G is a group generated by a finite set X. The first step is to 
choose and fix a particular way to describe finitely generated subgroups H of 
G. For example, a description S oi H could be a tuple of words (ui, . . . , Ufc) in 
the alphabet X^^ ^ X U X^^ representing a set of generators of H, or a set 
of words {ui, . . . ,Uk} that generates H, or a folded finite graph that accepts 
the subgroup generated by the generators {ui, . . . , Uk} of H in the ambient free 
group F{X) (see [19), etc. In general, the descriptions above, by no means are 
unique for a given subgroup iJ, in fact, we listed them here in the decreasing 
degree of repetition. 

When the way to describe subgroups in G is fixed one can consider the set 
A of all such descriptions of all finitely generated subgroups of G. The next 
step is to define a size s{5) of a given description (5 g A, i.e., a function 

s : A ^ N 

in such a way that the set (the ball of radius n) 

Bn = {5 e A I s{S) < n} 

is finite. This gives a stratification of the set A into a union of finite balls: 

A = U- iBn. (1) 

Let fin be a given probabilistic measure on _B„ (it could be the measure in- 
duced on Bn by some fixed measure on the whole set A or a measure not related 
to any measure on A). The stratification ([1]) and the ensemble of measures 

{^^n} = {^^n I neN} (2) 

allow one to estimate the asymptotic behavior of subsets of A. For a subset 
i? C A the asymptotic density Pfi{R) is defined by the following limit (if it 
exists) 

p,,{R) = Van Hn{RnBn). 
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If fin is the uniform distribution on the finite set i?„ then 



/i„(i?US„) 



|i?ns, 

\Bn\ 



n 



is the n-th frequency, or probabihty, to hit an element from R in the baU i3„. 
In this case we refer to p^{R) as to the asymptotic density of R and denote it 



One can also define the asymptotic densities above using lim sup rather then 
lim, in which event p^{R) does always exist. 

We say that a subset i? C A is generic if p^{R) = f and negligible if p^i{R) = 
0. It is worthwhile to mention that the asymptotic densities not only allow one 
to distinguish between "large" (generic) and "small" (negligible) sets, but give 
a tool to differentiate between various large (or small) sets. For instance, we 
say that R has asymptotic density p^(i?) with a super-polynomial convergence 
rate if 



for any fc G N. For brevity, R is called strongly generic if (i?) = 1 with a super- 
polynomial convergence rate. The set R is strongly negligible if its complement 
5* — i? is strongly generic. 

Similarly, one can define exponential convergence rates and exponentially 
generic (negligible) sets. 

2.2 Random subgroups and generating tuples 

In this section we follow the most commonly used in cryptography procedure to 
generate random subgroups of a given group (see for example P]). In brief, the 
following procedure is often employed: 

Random Generator of subgroups in G: 

• pick a random fc G N between given boundaries Kf) < k < Ki ; 

• pick randomly k words wi, . . . ,Wk G F{X) with fixed length range Lq < 

\wi\ < Li; 

• output a subgroup {wi, . . . , Wk) of G. 

Without loss of generality we may fix from the beginning a single natural number 
fc, instead of choosing it from the finite interval [Kq,Ki] (by the formula of 
complete probability the general ease can be reduced to this one). Fix fc G N, 
fc > 1, and focus on the set of all fc-generated subgroups of G. 

The corresponding descriptions (5, the size function, and the corresponding 
stratification of the set of all descriptions can be formalized as follows. By a 
description S{H) of a fc-generated subgroup _ff of G we understand here any 
fc-tuple {wi, . . . , Wk) of words from F{X) that generates H in G. Hence, in this 
case the space of all descriptions is the cartesian product F(X)'^ of fc copies of 



by p{R). 




F{X): 



A = Afc = F{X)K 
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The size s{'Wi, . . . ,Wk) can be defined as the total length of the generators 

S{wi, ...,Wk,) = \wi\ + . . . + \Wk\, 

or as the maximal length of the components: 

s{wi, . . . , Wfe) = max{|wi|, . . . , \wk\}. 

Our approach works for both definitions, so we do not specify which one we use 
here. For n g N denote by Bn the ball of radius n in A: 

S„ = {(wi,...,u'fe) g F{X)'' I s(wi,...,Wfc) < n}. 

This gives the required stratification 

For a subset M of A we define the asymptotic density p{AI) relative to the 
stratification above assuming the uniform distribution on the balls -B„: 

piM)^ hm 



\Bn\ 



Notice, that there are several obvious deficiencies in this approach: we consider 
subgroups with a fixed number of generators, every subgroup may have distinct 
fc-generating tuples, every generator can be described by several distinct words 
from F{X), i.e., our descriptions are far from being unique. However, as we 
have mentioned above, this models describe the standard methods to generate 
subgroups in cryptographic protocols. We refer to [36] for other approaches. 



2.3 Asymptotic properties of subgroups 

Let G be a group with a finite set of generators X and k a fixed positive natural 
number. Denote by 7-" a property of descriptions of fc-generated subgroups of 
G. By V{G) we denote the set of all descriptions from A = A^ that satisfy V 
in G. 

Definition 2.1. We say that a property P C A of descriptions of fc-generated 
subgroups of G is: 

1) asymptotically visible in G if p{'P{G)) > 0; 

2) generic in G if p{V{G)) = 1; 

3) strongly generic in G if p{'P{G)) ~ 1 and the rate of convergence of 
p„{V{G)) is super-polynomial; 

4) exponentially generic in G if p{'P{G)) = 1 and the rate of convergence of 
Pn{'P{G)) is exponential. 
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Informally, if V is asymptotically visible for fc-generated subgroups of G 
then there is a certain non-zero probability that a randomly and uniformly 
chosen description (5 G A of a sufficiently big size results in a subgroup of G 
satisfying V . Similarly, if V is exponentially generic for fc-generated subgroups 
of G then a randomly and uniformly chosen description i5 € A of a sufficiently 
big size results in a subgroup of G satisfying V with overwhelming probability. 
Likewise, one can interpret generic and strongly generic properties of subgroups. 
If a set of descriptions A of subgroups of G is fixed, then we sometimes abuse 
the terminology and refer to asymptotic properties of descriptions of subgroups 
as asymptotic properties of the subgroups itself. 

Example 2.2. Let if be a fixed fc-generated group. Consider the following 
property Vh'- a given description [wi, . . . ,Wk) G F{XY satisfies Vh if the 
subgroup (wi, . . . , Wfc), generated in G by this tuple, is isomorphic to H. If 
Vh{G) is asymptotically visible (generic) in A then we say that the group H is 
asymptotically visible (generic) in G (among fc-generated subgroups). 

By k-spectrum Speck{G) of G we denote the set of all (up to isomorphism) 
fc-generated groups which are asymptotically visible in G. 

There are several natural questions about asymptotically visible subgroups 
of G that play an important part in cryptography. For example, when choosing 
fc-generated subgroups of G randomly it might be useful to know what kind of 
subgroups you can get with non-zero probability. Hence the following question 
is of interest: 

Problem 2.3. What is the spectrum SpeCk{G) for a given group G and a 
natural number fc > 1? 

More technical, but also important in applications is the following question. 

Problem 2.4. Does the spectrum Speck{G) depend on a given finite set of 
generators of G? 

We will see in due course that answers to these questions play an important 
part in the choice of strong keys in some group-based cryptosystems. 

2.4 Groups with generic free basis property 

Definition 2.5. We say that a tuple (ui,...,Ufc) G F{X)^ has a free basis 
property (J-B) in G if it freely generates a free subgroup in G. 

In [27] Jitsukawa showed that is generic for fc-generated subgroups of 
a finitely generated non-abelian group F{X) for every fc > 1 with respect to 
the standard basis X. Martino, Turner and Ventura strengthened this result in 
[35] . they proved that is exponentially generic in F{X) for every fc > 1 with 
respect to the standard basis X. Recently, it has been shown in [35] that TB 
is exponentially generic in arbitrary hyperbolic non-elementary (in particular, 
free non-abelian) group for every fc > 1 and with respect to any finite set of 
generators. 
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We say that the group G has the generic free basis property if is generic 
in G for every fc > 1 and every finite generating set of G. Similarly, we define 
groups with strongly and exponentially generic free basis property. By TBgen, 
TBst, J'Bexp we denote classes of finitely generated groups with, correspond- 
ingly, generic, strongly generic, and exponentially generic, free basis property. 

The following result gives a host of examples of groups with generic TB. 

Theorem 2.6. Let G he a finitely generated group and N a normal subgroup 



of G. If the quotient group G/N is in TBgen, 
the whole group G is in the same class. 



or in TBst, or in J- Be 



then 



Proof. Let H = G/N and ^ : G — > 7? be the canonical epimorphism. Fix a 
finite generating set X of G and a natural number A: > 1. Clearly, X'^ is a 
finite generating set of H. By our assumption, the free basis property is generic 
in H with respect to the generating set X'^ and given k. Identifying x & X 
with x'^ € H wc may assume that a finitely generated subgroup A of G and 
the subgroup A'^ have the same set of descriptions. Observe now, that for a 
subgroup ^ of G generated by a fc-tuple (ui, . . . , u^) G F(X)'^ if A'^' is free with 
basis {uf, . . . , uf,) then A is also free with basis (ui, . . . , Uk). Therefore for each 
t e N 

\BtnTB{G)\ ^ \Btr\TB{H)\ 



\B, 



> 



This implies, that if !FB{H) is generic in H = G/N, that J^B{G) is also generic in 
G, and its convergence rate in G is not less then the corresponding convergence 
rate in H , as claimed. 

□ 

The result above bears on some infinite groups used recently in group-based 
cryptography. Braid groups Bn appear as the main platform in the braid-group 
cryptography (see [21 [311 [iTl [3] ) . Recall that the braid group Bn can be defined 
by the classical Artin presentation: 



Bn = 



CTi, 



if K-j| = l 
if \i~j\ > I 



Denote by cr^.i+i the transposition (i, i+1) in the symmetric group S„. The map 
Ci — > CTi.i+i; i — 1, . . . ,n gives rise to the canonical epimorphism tt : _B„ I]„. 
The kernel of tt is a subgroup of index n\ in i?„, termed the pure braid group 

PBn. 

Corollary 2.7. The free basis property is exponentially generic in the pure 
braid groups PBn for n > 3. 

Proof. It is known (see |S], for example) that a pure braid group PB„,n > 3, 
has the group PB^, as its epimorphic quotient, and the group PB^ is isomorphic 
to F2 X Z, so PBn,n > 3, has the free group F2 as its quotient. Now, the 
result follows from Theorem 12 . 61 and the strong version of the Jitsukawa's result 

IMllSillll- □ 
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As we have seen a pure braid group PBn,n > 3, has exponentially generic 
free basis property and it is a subgroup of finite index in the braid group i3„. 
However, at the moment, wc do not have a proof that Bn has exponentially 
generic free basis property. Though, we conjecture that this should be true. 

Problem 2.8. Is it true that the braid groups n > 3, has exponentially 
generic free basis property? 

In [51| partially commutative groups were proposed as possible platforms 
for some cryptosy stems. We refer to [8] for more recent discussion on this. By 
definition a partially commutative group G{T) (also called, sometimes, as right 
angled Artin groups, or graph groups, or trace groups) is a group associated 
with a finite graph T = {V, E), with a set of vertices V = {wi, . . . , w„} and a set 
of edges E C V x V, hy the following presentation: 

G(r) = (wi, . . . ,i;„ I ViVj = VjV^ for {vi,v-j) £ E). 

Observe, that the group G{T) is abclian if and only if the graph F is complete. 

Corollary 2.9. The free basis property is exponentially generic in non-abclian 
partially commutative groups. 

Proof. Let G = G(r) be a non-abelian partially commutative group correspond- 
ing to a finite graph T. Then there are three vertices in F, say vi,V2,V3 such 
that the complete subgraph Tq of F generated by these vertices is not a triangle. 
In particular, a partially commutative group Go = G(Fo) is either a free group 
F3 (no edges in Fq), or (Z x Z) *Z (only one edge in Fq), or F2 x Z (precisely two 
edges in Fq). Notice that in all three cases the group G(Fo) has F2 as its epi- 
morphic quotient. Now, it suffices to show that G(Fo) is an cpimorphic quotient 
of G(r), which is obtained from G(F) by adding to the standard presentation of 
G(r) all the relations of the type v = 1, where u is a vertex of F different from 
Vi,V2,V3. This shows that F2 is a quotient of G(r) and the result follows from 
Theorem [221 □ 

Observe, that some other groups, that have been proposed as platforms in 
based-group cryptography, do not have non-abelian free subgroups at all, so they 
do not have free basis property for k > 2. For instance, in [44] the Grigorchuk 
groups were used as a platform. Since these groups are periodic (i.e., every 
element has finite order) they do not contain non-trivial free subgroups. It is 
not clear what are asymptotically visible subgroups in Grigorchuk groups. As 
another example, notice that in [46] authors put forth the Thompson group F 
as a platform. It is known that there are no non-abelian free subgroups in F 
(see, for example, [E]), so F does not have free basis property. Recently, some 
interesting results were obtain on the spectrum Speck{F) in [20] . 

2.5 Quasi-isometrically embedded subgroups 

In this section we discuss another property of subgroups of G that plays an 
important part in our cryptanalysis of group based cryptosystems. 
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Let G be a group with a finite generating set X. The Cayley graph T{G, X) 
is an X-labeled directed graph with the vertex set G and such that any two 
vertices g,h € G are connected by an edge from g to h with a label x £ X if and 
only ii gx ^ h in G. For convenience we usually assume that the set X is closed 
under inversion, i.e., x~^ G X for every x G X. One can introduce a metric dx 
on G setting dx {g, h) equal to the length of a shortest word in X^'^ ~ XlJ X~^ 
representing the clement g~^h in G. It is easy to see that dx{g,h) is equal to 
the length of a shortest path from g to h in the Cayley graph T{G,X). This 
turns G into a metric space (G, dx)- By lx{g) we denote the length of a shortest 
word in generators X^^ representing the element g, clearly lx{g) = dx{^,g)- 

Let H he a. subgroup of G generated by a finite set of elements Y. Then 
there are two metrics on H: the first one is dy described above and the other 
one is the metric dx induced from the metric space (G, dx ) on the subspace 
H. The following notion allows one to compare these metrics. Recall that a 
map / : Ah — > M2 between two metric spaces (Mi,di) and (M2,d2) is a quasi- 
isometric embedding if there are constants A > l,c > such that for every 
elements x,y £ Mi the following inequalities hold: 

jdi{x,y) - c < d2{f{x),f{y)) < Xdi{x,y) + c. (3) 

In particular, we say that a subgroup H with a finite set of generators Y is 
quasi-isometrically embedded into G if the inclusion map i : if ^ G is a quasi- 
isometric embedding i : {H^dy) {G,dx)- Notice, that in this case the right- 
hand inequality in ([3]) always holds, since for all f^h £ H 

dx{i{f),i{h)) < niax{lx{y)} ■ dY{f,h). 

yeY 

Therefore, the definition of quasi-isomctrically embedded subgroup takes the 
following simple form (in the notation above). 

Definition 2.10. Let G be a group with a finite generating set X and H 
a subgroup of G generated by a finite set of elements Y. Then H is quasi- 
isom,etrically embedded into G if there are constants A > 1, c > such that for 
every elements f,h £ H the following inequality holds: 

jdy{f,h)~C<dx{f,h). (4) 

It follows immediately from the definition, that if X and X' are two finite 
generating sets of G then the metric spaces {G,dx) and {G,dx') are quasi- 
isometrically embedded into each other. This implies that the notion of quasi- 
isometrically embedded subgroups is independent of the choice of finite gener- 
ating sets in H or in G (though the constants A and c could be different). 

Definition 2.11. Let G be a group with a finite generating set X. We say that 
a tuple (wi, . . . ,Uk) € F{X)^ has a QI (quasi-isometric embedding) property 
in G if the subgroup it generates in G is quasi-isometrically embedded into G. 
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Denote by QX{G) the set of all tuples in F{X)'' that satisfy the QI prop- 
erty in G. We term the property QI is generic in G if QI{G) is generic in 
G for every fc > 1 and every finite generating set of G. Similarly, we de- 
fine groups with strongly and exponentially generic quasi-isometric embedding 
subgroup property. Denote by QIgen, QJ-st-, QJ-ex-p classes of finitely gener- 
ated groups with, correspondingly, generic, strongly generic, and exponentially 
generic, quasi-isometric embedding subgroup property. 

It is not hard to see that every finitely generated subgroup of a finitely 
generated free group F is quasi-isometrically embedded in F, so F G QJexp- 

The following result gives further examples of groups with quasi-isometric 
embedding subgroup property. 

Let G G TBgen n QJgen- Noticc, that the intersection of two generic sets 
J^B(G) C and QI{G) C F{X)'' is again a generic set in so 

the set J^B{G) n QX(G) of aU descriptions (ui, . . . ,Ufc) £ that freely 

generate a quasi-isometrically embedded subgroup of G, is generic in F{X)''. 
Observe, that by the remark above, and the result on free basis property in free 
groups, J-Bgen n QXgen coutains all free groups of finite rank. The argument 
applies also to the strongly generic and exponentially generic variations of the 
properties. To unify references we will use the following notation: TB^ fl QX* 
for * G {gen, st, exp}. 

Theorem 2.12. Let G be a finitely generated group with a quotient G/N . If 
G/N E TB^ n QI* then G G TB* n QI* for any * G [gen, st, exp}. 

Proof. Let G be a finitely generated group generated hy X, N a normal sub- 
group of G such that the quotient G/N is in TB* n QI*. Let (j) : G ^ G/N 
be the canonical epimorphism. By Thcorem l2.6I G G J-'B*, so it suffices to show 
now that G G QI* . 

Let H he a A:-generated subgroup with a set of generators Y ~ {ui, . . . , u^,) G 
Suppose that Y G TB*{G/N) n QI*{G/N), i.e., the image F"^ of Y 
in G/N freely generates a free group quasi-isometrically embedded into G/N. 
Observe, first, that for every element g G G one has lx{g) > ^X't-ig'^): where lx<t> 
is the length on G/N relative to the set of generators X'^ . Since the subgroup 
H'^ is quasi-isometrically embedded into G/N the metric space {W^ , dy*) quasi- 
isometrically embeds into (G^, dx*)- On the other hand, maps the subgroup 
H onto the subgroup H'^ isomorphically (since both are free groups with the 
corresponding bases), such that for any h £ H dyih) = dypih'^). Now we can 
deduce the following inequalities for g,h E H: 

jdvig, h)~c^ ^dy, (5^ h*)~c< dx, (<?^ h*) < dx {g, h) 

where A and c come from the quasi-isometric embedding of H"^ into G/N. This 
shows that H is quasi-isometrically embedded into G, as required. 

□ 

Corollary 2.13. The following groups are in J^Bexp n Qlexp- 
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1) Pure braid groups PBn, n > 3; 

2) Non-abclian partially commutative groups G{T). 

Proof. The arguments in Corollaries l2.7[ [^T^ show that the groups PBn, n > 3, 
and G(r), non-commutative, have quotient isomorphic to the free group i^2- 
Now the result follows from Theorems 12.61 and 12.121 □ 

3 Anshel-Anshel-Goldfeld scheme 

In this section we discuss the Anshel-Anshel-Goldfeld (AAG) cryptosystem for 
public key exchange [5] and touch briefly on its algorithmic security. 

3.1 Description of Anshel-Anshel-Goldfeld scheme 

Here we give a general description of the Anshel-Anshel-Goldfeld cryptosystem. 

Let G be a group with a finite generating set X, it is called the platform 
of the scheme. We assume that elements w in G have unique normal forms w 
such that it is "hard" to reconstruct w from w and there is a "fast" algorithm 
to compute u* when given w. We do not discuss here the security issues of these 
two components of the platform G, leaving this for the future. 

The Anshel-Anshel-Goldfeld key exchange protocol requires the following 
sequence of steps. Alice [Bob resp.] chooses a random subgroup of G 

A = (ai , . . . , a,„) [B = {bi, . . . ,bn) resp.] 

by randomly choosing generators ai, . . . , [6i, . . . , 6„ resp.] as words in X^^, 
and makes it public. Then Alice [Bob resp.] chooses randomly a secret element 
a = «(«!,..., a,„) G yl [6 = w(6i,...,5„) e B resp.] as a product of the 
generators of A [B resp.] and their inverses, takes the conjugates 
[a\, . . . , resp.], encodes them by taking their normal forms 6f [aj resp.], and 
makes these normal forms public: 

[a5,...,a^ resp.]. 
Afterward, they both can compute the secret shared element of G: 

a-\^ = [a, 6] = {b^y^h 
and take its normal form as the secret shared key. 

3.2 Security assumptions of AAG scheme 

In this section we briefly discuss computational security features of the AAG 
cryptosystem. Unfortunately, in the original description of AAG the authors 
did not state precisely what are the security assumptions that make the system 
difficult to break. Here we dwell on several possible assumptions of this type. 
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that often occur, though sometimes hiiphcitly, in the literature on the AAG 
cryptosystem. 

It seems that the security of AAG relies on the computational hardness of 
the following, relatively new, computational problem in group theory: 

AAG Problem: given the whole public information from the scheme AAG, 
i.e., the group G, the elements ai, . . . , a,m, bi, . . . , bn, and bf,. . . ,6^,aJ, ■■ ■,a\ 
in a group G, find the shared secret key [a, 6]. 

This problem is not a standard group-theoretic problem, not much is known 
about its complexity, and it is quite technical to formulate. So it would be 
convenient to reduce this problem to a standard algorithmic problem in groups 
or to a combination of such problems. The following problems seem to be 
relevant here and they attracted quite a lot of attention recently, especially in 
the braid groups - the original platform for AAG [2] . We refer to papers [11] , [6] , 
[7], [22], [33], [34]. Nevertheless, the precise relationship between these problems 
and AAG is unclear, see [IT] for more details. 

The Conjugacy Search Problem (CSP): given u, v G G such that an equa- 
tion = V has a solution in G, find a solution. 

The Simultaneous Conjugacy Search Problem (SCSP): given u^, Vi G G, 
such that a system uf — Vi^ i = 1^ . . . ,n has a solution in G, find a solution. 

The Simultaneous Conjugacy Search Problem relative to a subgroup 
(SCSP*): given Ui, vi G G and a finitely generated subgroup A of G such that 
a system itf = u^, i = 1, . . . , n has a solution in A, find such a solution. 

Remark 3.1. Observe, that if the Word Problem is decidable in G then all 
the problems above are also decidable. Indeed, one can enumerate all possible 
elements x (either in G or in the subgroup A) , substitute them one-by -one into 
the equations, and check, using the decision algorithm for the Word Problem in 
G, if a; is a solution or not. Since the systems above have some solutions this 
algorithm will eventually find one. However, the main problem here is not about 
decidability, the problem is whether or not one can find a solution sufficiently 
"quickly", say in polynomial time in the size of the inputs. 

The following result is easy. 

Lemma 3.2. For any group G the AAG problem can be reduced in linear time 
to the problem SCSP*. 

Proof. Suppose in a finitely generated group G we are given the public data 
from the AAG scheme, i.e., the subgroups 

A= {ai,. . . ,am), B = . . . ,&„), 
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and the elements 6" and a\, . . . ,a^^. If the problem SCSP relative to 

subgroups A and B is decidable in G, then solving a system of equations 

bl = bt,...,bl=bl (5) 

in A one can find a solution u G A. Similarly, solving a system of equations 

of = a?,---,< =a?„ (6) 

in B one can find a solution v & B. Notice, that all solutions of the system ^ 
are elements of the form ca where c is an arbitrary element from the centralizer 
Cg{B), and all solutions of the system ([6]) arc of the form db for some d £ Cg{A). 
In this case, obviously [u, v] = [ca, db] = [a, b] gives a solution to the the AAG 
problem. □ 

Clearly, in some groups, for example, in abclian groups AAG problem as 
well as the SCSP* are both decidable in polynomial time, which makes them 
(formally) polynomial time equivalent. We will sec in Section that SCSP* 
is easy in free groups. 

It is not clear, in general, whether the SCSP is any harder or easier than the 
CSP. In hyperbohc groups SCSP, as well as CSP, is easy [TT] . 

There are indications that in finite simple groups, at least generically, the 
SCSP* is not harder than the standard CSP (since, in this case, two randomly 
chosen elements generate the whole group). We refer to a preprint [24] for a 
brief discussion on complexity of these problems. 

It is interesting to get some information on the following problems, which 
would shed some light on the complexity of AAG problem. 

Problem 3.3. 1) In which groups AAG problem is poly-time equivalent to 
the SCSP*? 

2) In which groups SCSP* is harder than the SCSP? 

3) In which groups SCSP is harder (easier) than CSP? 

In the rest of the paper we study the hardness of SCSP* in various groups 
and analyze some of the most successful attacks on AAG from the view-point 
of asymptotic mathematics. 



4 Length Based Attacks 

The intuitive idea of the length based attack (LBA) was first put in the paper 
[26] by J. Hughes and A. Tanncnbaum. Later it was further developed in a joint 
paper |23| by Garber, Kaplan, Tcicher, Tsaban, and Vishne where the authors 
gave an experimental results concerning the success probability of LBA that 
suggested that very large computational power is required for this method to 
successfully solve the Conjugacy Search Problem. 
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Recently, the most successful variation of this attack for braid groups was 
developed in [39] where the authors suggested to use a heuristic algorithm for 
approximation of the geodesic length of braids in conjunction with LBA. Fur- 
thermore, the authors analyzed the reasons for success/failure of their variation 
of the attack, in particular the practical importance of Alice's and Bob's sub- 
groups A and B being non isometrically embedded and being able to choose the 
elements of these subgroups distorted in the group (they refer to such elements 
as peaks). 

In this section we rigorously prove that the same results can be observed 
in much larger classes of groups. In particular our analysis works for the class 
J-Bexp and, hence, for free groups, pure braid groups, locally commutative non- 
abelian groups, etc. 

4.1 A general description 

Since LBA is an attack on AAG scheme the inputs for LBA are precisely the 
inputs for AAG algorithmic problem. Moreover, in all its variations LBA attacks 
AAG via solving the corresponding conjugacy equations given in a particular 
instance of AAG. In what follows we take a slightly more general approach and 
view the length based attack (LBA) as a correct partial search deterministic 
algorithm of a particular type for the Simultaneous Conjugacy Search Problem 
relative to a subgroup in a given group G. In this case LBA is employed to solve 
SCSP*, not AAG. Below we describe a basic LBA in its most simplistic form. 

Let G be a group with a finite generating set X. Suppose we are given a 
particular instance of the SCSP*, i.e., a system of conjugacy equations uf = 
Vi^i = 1, . . . ,TO which has a solution in a subgroup A = (Y) generated by a 
finite set Y of elements in G (given by words in F{X)). The task is to find such 
a solution in A. The main idea of LBA is very simple and it is based on the 
following assumptions: 

(LI) for arbitrary "randomly chosen" elements u,w £ G one has Ix^u"^) > 
Ix (u); 

(L2) for "randomly chosen" elements w, j/i, . . . , ?/fc in G the element w has min- 
imal Zx-length among all elements of the type , where y runs over the 
subgroup of G generated by yi , . . . , . 

It is not obvious at all whether this assumption is realistic or not, or even how to 
formulate it correctly. We will return to these issues in due course. Meantime, 
to make use of the assumptions above we assume that we are given an algorithm 
A to compute the length function lx{w) for a given element w € G. 

Consider Alice' public conjugates . . . , where a = a^J . . . a^^. Essen- 
tially each bf is a result of a sequence of conjugations of bi by the factors of 
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A: 

h 

I 

a7'' b, all 
I 

ar/^a.T^ h allall (7) 

i 

I 

A conjugating sequence is the same for each bi and is defined by the private key 
a. The main goal of the attack is to reverse the sequence ([7]) and going back 
from the bottom to the top recover each conjugating factor. If successful the 
procedure will result in the actual conjugator as a product of elements from a. 

The next algorithm is the simplest realization of LBA called the best descend 
LBA. It takes as an input three tuples (ai, . . . , a,n), . . . , bn), and (ci, . . . , c„) 
where the last tuple is assumed to be 6°, . . . , The algorithm is a sequence of 
the following steps: 

— (Initialization) Put x = e. 

— (Main loop) For each i ~ l,...,n and £ = ±1 compute k^e = 
Ej=i^^(ar^Cjaf). 

— If for each i = 1, . . . , rt and e — ±1 the inequality > ^x{cj) 
is satisfied then output x. 

— Otherwise pick i and e giving a least value Z^.g. Multiply x on the 
right by af. For each j = 1, . . . , n conjugate Cj = a^^Cjaf. Continue. 

— (Last step) If Cj = bj for each j = 1,. . . ,n then output the obtained 
element x. Otherwise output Failure. 

Other variations of LBA suggested in [39] are LBA with Backtracking and Gen- 
eralized LBA. We refer to [39] for a detailed discussion on this. 

One can notice that instead of the length function Ix one can use any other 
objective function satisfying assumptions (LI) and (L2). In this work besides 
Ix we analyze the behavior of modifications of LBA relative to the following 
functions: 

(Ml) Instead of computing the geodesic length lx{vi) of the clement Vi G G 
compute the geodesic length Izi^i) in the subgroup H generated by Z = 
{u} U Y (clearly, Vi e H). In this case, LBA in G is reduced to LBA in 
H , which might be easier. We term Iz the inner length in LBA. 

(M2) It might be difficult to compute the lengths lx{w) or lz{w)- In this case, 
one can try to compute some "good", say linear, approximations of Ixiw) 
or lz{w), and then use some heuristic algorithms to carry over LBA (see 

m)- 
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These modifications can make LBA mucli more efficient as we will see in the 
sequel. 

In what follows our main interest is in the generic time complexity of LBA. 
To formulate this precisely one needs to describe the set of inputs for LBA and 
the corresponding distribution on them. 

Recall that an input for SCSP* in a given group G with a fixed finite gener- 
ating set X consists of a finitely generated subgroup A ~ {ai, . . . , ak) of G given 
by a fc-tuple (oi, . . . , Ofc) G F{X)'', and a finite system of conjugacy equations 

= Vi, where Ui, vi E F{X), z = 1, . . . , m, that has a solution in A. We denote 
this data by a = (T, b), where T = (ai, . . . , Ofe, mi, . . . , Um) and b = (wi, . . . , Vm)- 
The distinction that we make here between T and b will be in use later on. For 
fixed positive integers m, k we denote the set of all inputs a ~ (T, b) as above 
by hjn- 

The standard procedure to generate a "random" input of this type in AAG 
protocol is as follows. 

A Random Generator of inputs for LBA in a given G: 

• pick a random fc G N from a fixed interval Kq < k < Ki] 

• pick randomly k words ai, . . . , e F{X) with the length in fixed interval 
Lo < < Li, 

• pick a random m G N from a fixed interval Mq < m < Mi] 

• pick randomly m words ui, . . . , Um G F{X) with the length in fixed inter- 
val No < \u,\ < Ni; 

• pick a random element w from the subgroup A = (ai, . . . , ak), as a random 
product w = Qi-^Qi^ . . . of elements from {ai, . . . , ak} with the number 
of factors c in a fixed interval Pi < c < P-z; 

• conjugate Vi = uf and compute the normal form of w,, z = 1, . . . , m. 

As we have argued in Section 12.21 one can fix the numbers k, m, and the 
number of factors c in the product w, in advance. Observe, that the choice 
of the elements vi, . . . , Vm is completely determined by the choice of the tuple 
T = (oi, . . . , Ofc, ui, . . . , Um) G F(Ar)'^+™ and the word w. 

Notice, that the distribution on the subgroups H = (T) (more precisely, 
their descriptions from F(A')'^"'"™) that comes from the random generator above 
coincides with the distribution on the (fc 4- m)-generatcd subgroups (their de- 
scriptions) that was described in Section [2T2l We summarize this in the following 
remark. 

Remark 4.1. 1) The choice of a tuple T = (oi, . . . , Ofc, mi, . . . , Wm) € 
F(A')*''+™ precisely corresponds to the choice of generators of random 
subgroups described in Section [2T2l 

2) Asymptotic properties of the subgroups generated by T precisely corre- 
spond to the asymptotic properties of subgroups discussed in Section [2] 
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4.2 LBA in free groups 



In this section we discuss LBA in free groups. It is worthwhile to mention here 
that there are fast (quadratic time) algorithms to solve SCSP* and, hence, AAG 
in free groups (see Section However, results on LBA in free groups will 

serve us as a base for solving SCSP* in many other groups. 

Let A: be a fixed positive natural number. We say that cancelation in a set 
of words Y = {yi, . . . , yk} Q F{X)'' is at most A, where A S (0, 1/2), if for any 
u,w £ Y^^ the amount of cancelation in the product uv is strictly less then 
Xmm{lx{u),lx{v)}, provided u ^ in F{X). 

Lemma 4.2. // the set Y = {yi, . . . ,yk} satisfies X-condition for some A £ 
(0,1/2) then: 

• The set Y is Nielsen reduced. In particular, Y freely generates a free 
subgroup and any element w G (Y) can be uniquely represented as a reduced 
word in the generators Y and their inverses. 

• The Membership Search Problem for a subgroup (Y) (see Section \6.1\ for 
details) is decidable in linear time. 

• The geodesic length for elements of a subgroup (Y) ( see Section I5.il for 
details) is computable in linear time. 

Proof. Easy exercise. 



Moreover, the following result is proved in [35j . 

Theorem 4.3. Let A £ (0, 1/2). The set S of k-tuples (m, . . .,Uk) £ F{X)'' 
satisfying X-condition is exponentially generic and, hence, the set of k-tuples 
which are the Nielsen reduced in F(X) is exponentially generic. 

Now wc arc ready to discuss the generic complexity of LBA in free groups. 

Theorem 4.4. Let F{X) be a free group with basis X. Then LBA with respect 
to the inner length Iz solves SCSP* in linear time on an exponentially generic 
set of inputs. 

Proof. Let n and m be fixed positive integers. Denote by 5 a set of (n + m)- 
tuples (mi, . . . , Un, ai, . . . , am) S i^(X)"+™ that satisfy 1/4-condition. It follows 
from Theorem 14.31 that the set S is exponentially generic. 

Furthermore, the system of conjugacy equations associated with such a tuple 
Z = (mi, . . . , m„, oi, . . . , a„) has the form 



where Vi belong to the subgroup (Z) generated by Z and x is searched in the 
same subgroup. By Lemma 14.21 one can find expressions for Vi in terms of the 



□ 
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generators Z in linear time. Now, since the generators ai, . . . , a™ are part of the 
basis of the subgroup {Z) it follows that LBA relative to Iz successfully finds a 
solution X — w{ai, . . . , Um) hi linear time. 

□ 

4.3 LBA in groups from TBexp 

The result above for free groups is not very surprising because of the nature 
of cancelation in free groups. What, indeed, looks surprising is that LBA 
works generically in some other groups which seem to be very different from 
free groups. In this and the next section we outline a general mathematical 
explanation why LBA has a high rate of success in various groups, including 
the braid groups. In particular, it will be clear why Modification (Ml) of LBA, 
which was discussed in Section |4.H is very robust, provided one can compute 
the geodesic length in subgroups. 

We start with a slight generalization of the result of Theorem 14.41 Recall 
(from Section H?T|) that inputs for LBA, as well as for SCSP*, can be described 
in the form a = {T, b), where T = (oi, . . . , a^, mi, . . . , Um) £ F(X)'^+™ and 
b = (ui, . . . ,Vm), such that there is a solution of the system uf = Vi in the 
subgroup A — (fli, . . . , a^). 

Lemma 4.5. Let G be a group with a finite generating set X and Ik.rn o- set of 
all inputs (T, b) for LBA in G. Put 

I free = {{T,b) G Ik.m \ T freely generates a free subgroup in G}. 

Suppose there is an exponentially generic subset S of I free o,nd an algorithm 
A that computes the geodesic length It of elements from the subgroup {T), 
{T,b) G S, when these elements are given as words from F{X). Then there 
is an exponentially generic subset S' of Lfree such that on inputs from S' LBA 
halts and outputs a solution for the related SCSP* in at most quadratic time 
relative to the algorithm A. 

Proof. The result directly follows from Theorem 14.41 □ 

Let G e TBexp- In the next theorem we prove that the time complexity of 
SCSP* on an exponentially generic set of inputs is at most quadratic relative to 
the time complexity of the problem of computing the geodesic length in finitely 
generated subgroup of G. 

Theorem 4.6. (Reducibility to subgroup-length function) Let G be a 

group with exponentially generic free basis property and X a finite generating 
set of G. Then there is an exponentially generic subset S of the set Ik,m of all 
inputs for LBA in G such that on inputs from S LBA relative to It halts and 
outputs a solution for the related SCSP*. Moreover, the time complexity of LBA 
on inputs from S is at most quadratic relative to the algorithm A that computes 
the geodesic length It of elements from the subgroup (T) when these elements 
are given as words from F{X). 
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Proof. By Lemma [4.51 there is an exponentially generic subset S of I free such 
that on inputs from S LBA halts and outputs a solution for the related SCSP*. 
Moreover, the time complexity of LBA on inputs from S is at most quadratic 
relative to the algorithm A that computes the geodesic length It of elements 
from the subgroup (T) when these elements are given as words from F{X). It 
suffices to show now that the set Ifree is exponentially generic in the set of all 
inputs / for LBA in G. By Remark |4 . 1 1 asymptotic density of the set Ifree in I 
is the same as the asymptotic density of the set of tuples T G F{X)''~^"^ which 
have free basis property in G. Since G is in TBexp this set is exponentially 
generic in F{X)''~^™-, so is Ifree in /. This proves the theorem. □ 

5 Computing the geodesic length in a subgroup 

For groups G £ J-Bexp Theorem 14.61 reduces in quadratic time the time com- 
plexity of LBA on an exponentially generic set of inputs to the time complexity 
of the problem of computing the geodesic length in finitely generated subgroups 
of G. In this section we discuss time complexity of algorithms to compute the 
geodesic length in a subgroup of G. This discussion is related to Modification 2 
of LBA, introduced in Sect ion [4. II In particular, we focus on the situation when 
we do not have fast algorithms to compute the geodesic length of elements in 
finitely generated subgroups of G, or even in the group G itself. In this case, 
as was mentioned in Modification 2, one can try to compute some linear ap- 
proximations of these lengths and then use heuristic algorithms to carry over 
LBA. 

In Section [521 we discuss hardness of the problem of computing the geodesic 
length (GL problem) in braid groups i3„ - the original platforms of AAG pro- 
tocol. The time complexity of GLP in Bn relative to the standard set of Artin 
generators E is unknown. We discuss some recent results and conjectures in this 
area. However, there are efficient linear approximations of the geodesic length 
in Bn relative to the set of generators A (the generalized half-twists). Theo- 
retically, this gives linear approximations of the geodesic length of elements in 
Bn in the Artin generators, and, furthermore, linear approximations of geodesic 
inner length in quasi-isometrically embedded subgroups. If, as conjectured, the 
set of quasi-isometrically embedded subgroups is exponentially generic in braid 
groups, then this gives a sound foundation for LBA in braid groups. Notice, 
that even linear approximations alone are not entirely sufficient for successful 
LBA. To get a precise solution of SCSP* one needs also a robust "local search" 
near a given approximation of the solution. To this end several efficient heuristic 
algorithms have been developed [40], [39]. Nevertheless, by far none of them 
exploited directly the interesting interplay between geodesic lengths in S and 
A, as well as quasi-isometric embeddings of subgroups. 
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5.1 Related algorithmic problems 

We start with precise formulation of some problems related to computing 
geodesies in G. 

Computing the geodesic length in a group (GL): Let G be a group with 
a finite generating set X. Given an element u; G G, as a product of generators 
form X, compute the geodesic length lx{w). 

Computing the geodesic length in a subgroup (GLS): Let G be a group 
with a finite generating set X and A a subgroup of G generated by a finite set 
of elements Y = {ai, . . . , Ofc} of G given as words from F{X). Given an element 
w € A, as a product of generators of A, compute the geodesic length Iy{w). 

There is another (harder) variation of this problem, that comes from the 
SCSP* problem: 

Computing the geodesic length in a subgroup (GLS*): Let G be a group 
with a finite generating set X and A a subgroup of G generated by a finite set 
of elements Y = {ai, . . . , ak} of G given as words from F{X). Given an element 
w € A, as a word from F{X), compute the geodesic length Iy{w). 

The following lemma is obvious. Recall, that The Membership Search Prob- 
lem (MSP) for a subgroup ^ in G requires for a given element w S F(X), which 
belongs to A, to find a decomposition of w into a product of generators from Y 
and their inverses. 

Lemma 5.1. Let G be a finitely generated group and A a finitely generated 
subgroup of G. Then: 

1) GLS is linear time reducible to GLS*; 

2) GLS* is linear time reducible to GLS relative to the Membership Search 
Problem in A. 

Observe, that if GLS has a "fast" solution for ^ = G in G then there is a 
fast algorithm to find the geodesic length of elements of G with respect to X . 
In particular, the Word Problem in G has a fast decision algorithm. In some 
groups, like free groups or partially commutative groups, given by the standard 
generating sets, there are fast algorithms for computing the geodesic length 
of elements. In many other groups, like braid groups, or nilpotent groups, 
the computation of the geodesic length of elements is hard. Nevertheless, in 
many applications, including cryptography, it suffices to have a fast algorithm 
to compute a reasonable, say linear, approximation of the geodesic length of a 
given element. To this end we formulate the following problem. 

Computing a linear approximation of the geodesic length in a group 
(AGL): Let G be a group with a finite generating set X. Given a word 
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w £ F{X) compute a linear approximation of the geodesic length of w. More 
precisely, find an algorithm that for w £ F{X) outputs a word w' £ F{X) such 
that Xlxiw) + c > lx{w'), where A and c are independent of w. 

Another problem is to compute a good approximation in a subgroup of a 
group. 

Computing a linear approximation of the geodesic length in a sub- 
group (AGLS): Let G be a group with a finite generating set X and A a 
subgroup of G generated by a finite set of elements Y = {ai, . . . , Ofc} of G given 
as words from F{X). Given an element ui G A, as a word from F{X), compute 
a linear approximation of the geodesic length Iy{w) of w. 

Assume now that there is a "fast" algorithm to compute AGL in the group 
G. However, this does not imply that there is a fast algorithm to compute a 
linear approximation of the geodesic length in a given subgroup A of G. Unless, 
the subgroup A is quasi-isomctrically embedded in G. 

Lemma 5.2. Let G be a group with a finite generating set X and A is an 
algorithm to compute AGL in G with respect to X . If H is a quasi-isometrically 
embedded subgroup of G generated by a finite set Y then for every w S H , given 
as a word from F(X), the algorithm A outputs a word w' 6 F{X) such that 
Iy{w) < filx{w') + d for some constants fi and d which depend only on A and 



5.2 Geodesic length in braid groups 

There is no any known efficient algorithm to compute the geodesic length of 
elements in braid groups with respect to the set S of the standard Artin's 
generators. Some indications that this could be a hard problem are given in 
[43] , where the authors prove that the set of geodesies in Boo is co-NP-complete. 
However, in a given group, the problem of computing the length of a word 
could be easier then the problem of finding a geodesic of the word. Moreover, 
complexity of a set of geodesies in a group may not be a good indicator of 
the time complexity of computing the geodesic length in a randomly chosen 
subgroup. In fact, it has been shown in [40] |4T] that in a braid group Bn 
one can efficiently compute a reasonable approximation of the length function 
on Bn (relative to E) which gives a foundation for successful LBA. without 
computing the length in the group. Furthermore, there are interesting open 
conjectures that, if settled affirmatively, will lead to more efficient algorithms 
for computing the length of elements in braid groups and their subgroups. To 
explain this we need to introduce some known facts and terminology. 
The group i?„ has the classical Artin presentation: 



H. 



Proof. The proof is straightforward. 



□ 
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By l-s{w) we denote the length of a word w E Bn relative to the generating 
set S = {(Ti, . . . ,(T„_i}. 

Elements in _B„ admit so-called Garside normal forms. These forms are 
unique and the time complexity to compute the normal form of an element of 
Bn given by a word w S is bounded by 0(|wpn^). However, Garside 

normal forms are far from being geodesic in B„. 

In 1991 Patrick Dehornoy introduced in [14] the following notion of cr-positive 
braid word and a handle-reduction algorithm to compute a a-positive represen- 
tative of a given word. A braid word w is termed to be Ufc-positive (respectively, 
negative), if it contains ak, but does not contain ct^^ and a^'^ with i < k (re- 
spectively, contains cr^^, but not ak and af^ with i < k). A braid word w is 
said to be a-positive (respectively, cr-negative), if it is Cfe-positive (respectively, 
(Tfc-negative) for some k < n — 1. A braid word w is said to be a-consistent if it 
is either trivial or a-positive, or a-negative. 

Theorem. [Dehornoy |14|]. For any braid f3 6 Bn, exactly one of the 
following is true: 

1) (3 is trivial; 

2) (3 can he presented by a^-positive braid word for some k; 

3) (3 can be presented by a^-negative braid word for some k. 
In the latter two cases k is unique. 

Thus, it makes sense to speak about cr-positive and CT^-positive (or a-, cr-- 
negative) braids. 

The following question is of primary interest when solving AGL in braid 
groups: is there a polynomial p(x) such that for every word w G F{T?) p(l-s:{w)) 
gives an upper bound for the E-length of the shortest cr-consistent braid word 
representing w G i?ri? Dehornoy's original algorithms in [14] . and the handle 
reduction from [15]), and the algorithm from [21], all of them give only an 
exponential bound on the length of the shortest cr-consistent representative. 

In [inj (see also [151 [H] for a related discussion) Dynnikov and Wiest for- 
mulated the following 

Conjecture 5.3. There are numbers A, c such that every braid uu G B„ has a 
cr-consistent representative whose E-length is bounded linearly by the E-length 
of the braid. 

They also showed that the conjecture above has a positive answer if the 
E-lcngth of elements is replaced by the A-lcngth (relative to a set of generators 
A). 

The set of generators A consists of the braids Ay , 1 < « < j < which are 
the half- twists of strands i through j: 

A,j ~ (crj...crj_i)(crj...crj_2)---0'j. 

A is a generating set of Bn, containing the Artins generators cr^ = A^.i+i, and 
the Garside fundamental braid Ai„. The compressed A-length of a word w of 
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the form 

w = A''^. ...A''=- , 
where kt ^ and Ai^.^j ^ Aj^^j^ j^^^ for all t, is defined by 

La{w) =S,=ilog2(|fc.| + l). 

For an element /? G i?ri the value La{P) is defined by 

L/^(13) = min{LA(w) | t/ie word w represents 

Obviously, for any braid /3, we have 

La{I3) < Ia{P) < HP). 

The modified conjecture assumes the following extension of the notion of 
cr-positive braid word: a word in the alphabet A = {A^ | < z < j < n} is 
said to be cr-positive if, for some fc < Z, it contains A^i, and contains neither 
A^^^ nor A^^ with i < k and any j. In other words, a word w in letters Aij is 
cr-positivc (negative) if the word in standard generators a, obtained from w by 
the obvious expansion is. 

Theorem [Dynnikov, Wiest [19j]. Any braid (3 G Bn can he presented by a 
a-consistent word w in the alphabet {Ay} such that 

Ia{w) < SOnlAiP). 

This theorem gives a method to approximate geodesic length in braid groups, 
as well as in its quasi-isometrically embedded subgroups. It remains to be seen 
whether this would lead to more efficient versions of LBA or not. 

6 Quotient attacks 

In this section we describe a new type of attacks, which we term quotient attacks 
(QA). In fact, the quotient attacks are just fast generic algorithms to solve 
such search problems in groups as the Membership Search Problem (MSP), the 
Simultaneous Conjugacy Search Problem (SCSP), the Simultaneous Conjugacy 
Search Problem relative a to a subgroup (SCSP*), etc. The main idea behind 
QA is that to solve a problem in a group G it suffices, on most inputs, to solve 
it in a quotient G/N, provided G/N has generic free basis property and a fast 
decision algorithm for the problem. In particular, this is the case, if G has a 
free non-abelian quotient. Notice, that a similar idea was already exploited in 
P5] , but there the answer was given only for inputs in " No" part of the decision 
problem, which, obviously, does not apply to search problems. The strength of 
our approach comes from the extra requirement that G/N has the free basis 
property. 
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In Sections Wl] and we discuss the Conjugacy and Membership Problems 
in all their variations in free groups. Some of these results were known in 
folklore, some could be found in the literature. Nevertheless, we sketch most of 
the proofs here, since this will serve us as the base for solving similar problems 
in other groups. 

6.1 Membership Problems in free groups 

In this section wc discuss some algorithms to solve the Membership Problems 
in all their variations in free groups. We start with the classical Membership 
Problem (MP). Everywhere below G is a fixed group generated by a finite set 
X. 

The Membership Problem (MP): Let A = (ai, . . . , am) be a fixed finitely 
generated subgroup of G given by a finite set of generators ai, . . . , a™ (viewed 
as words in F{X)). Given a word w G F{X) decide whether w belongs to A or 
not. 

When the subgroup A is not fixed, but comes as a part of the input (like 
in AAG scheme) then the problem is more precisely described in its uniform 
variation. 

The Uniform Membership Problem (UMP): Given a finite tuple of ele- 
ments w, fli, . . . , am S F(X) decide whether or not w (viewed as an element of 
G) belongs to the subgroup A generated by the elements ai, . . . , am in G. 

To solve MP in free groups we use the folding technique introduced by 
Stallings in [48j, see also [28| for a more detailed treatment. Given a tuple 
of words oi, . . . , am G ^(^) one can construct a finite deterministic automaton 
Ta, which accepts a reduced word w G F{X) if and only if w belongs to the 
subgroup A = (ai, . . . , am) generated by ai, . . . , am in F{X). 

To describe the time complexity of MP and UMP recall that for a given 
positive integer n the function log2'n is defined as the least natural number m 
such that TO-towcr of exponents of 2 exceeds n, or equivalently, log2 o log2 o . . . o 
loQiin) < I, where on the left one has composition of m logarithms. 

Lemma 6.1. There exists an algorithm which for any input w,ai,...,am G 
F{X) for UMP finds the correct answer in nearly linear time 0(1^1 + nlog*n) 
where n ~ X]i=i l"^*!- Furthermore, the algorithm works in linear time 0{\w\+n) 
on exponentially generic set of inputs. 

Proof. Indeed, given w,ai, . . . , G F{X) one can construct Tj^ in worst time 
0{nlog*n) (see [33]) and check HVa accepts w or not in time 0(|w|), as required. 

To prove the generic estimate recall that the set of m-tuples ai, . . . , am G 
F{X) satisfying 1/4-condition is exponentially generic and the Stalling's proce- 
dure constructs the automaton Va in linear time 0{n). 

□ 
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In cryptography, the search variations of MP and UMP are the most inter- 
esting. 

The Membership Search Problem (MSP): Let A = (ai, . . . , am) be a fixed 
finitely generated subgroup of G given by a finite set of generators oi, . . . , Um, 
viewed as words in F{X). Given a word w G F{X), which belongs to A, find a 
representation of u; as a product of the generators ai, . . . , and their inverses. 

The Uniform Membership Search Problem (UMSP): Given a finite tu- 
ple of elements w,ai, . . . , a,„ G F{X) such that w ^ A ^ (ai, . . . , a^) find a 
representation of w as a product of the generators ai, . . . , and their inverses. 

Time complexity upper bounds for MSP easily follow from the corresponding 
bounds for MP. 

Lemma 6.2. The time complexity of MSP in a free group is bounded from above 
by 0{\w\). 

Proof. Let A = (ai, . . . ,am) be a fixed finitely generated subgroup of G. As 
was mentioned above in time 0{nlog*n), where n = |ai | -I- . . . -I- |a„|, one can 
construct the Stallings' folding F^. In linear time in n, using the breadth first 
search, one can construct a Nielsen basis S = {61, . . . , 6„} of A (see [H])- Now, 
given a word w G F{X), that belongs to A, one can follow the accepting path 
for w in Ta and rewrite i/j as a product of generators from S and their inverses. 
This requires linear time in jit; |. It is suffices to notice that the elements bi 
can be expressed as fixed products of elements from the initial generators of A, 
hi = Ui(ai, . . . , a„), i = 1, . . . , m, therefore any expression of w as a product of 
elements from S^^ can be rewritten in a linear time into a product of the initial 
generators. □ 

Observe, that in the proof above we used the fact that any product of new 
generators bi and their inversions can be rewritten in linear time into a product 
of the old generators Oi and their inversions. That held because we assumed 
that one can rewrite the new generators bi as products of the old generators Oj 
in a constant time. This is correct if the subgroup A is fixed. Otherwise, say 
in UMSP, the assumption does not hold anymore. It is not even clear whether 
one can do it in polynomial time or not. In fact, the time complexity of UMSP 
is unknown. The following problem is of prime interest in this area. 

Problem 6.3. Is the time complexity of UMSP in free groups polynomial? 

However, the generic case complexity of UMSP in free groups is known. 

Lemma 6.4. The generic case time complexity of UMSP in free groups is linear. 
More precisely, there is an exponentially generic subset T C F(Xy^ such that 
for every tuple {w,ai, . . . ,am) G F{X) xT, such that w G {ai,...,am), one 
can express w as a product o/oi, . . . , a„i and their inverses in time 0{\w\ + n) 
where n = |ai| + . . . + |a„|. 
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Proof. Notice, first, that if in the argument of Lemma 16.21 the initial set of 
generators ai, . . . , a,„ of a subgroup A satisfy 1/4-condition then the set of the 
new generators &i, ... ,6m coincides with the set of the initial generators (see 
P5] for details). Moreover, as was noticed in the proof of Theorem 14.41 the 
set T of tuples (ai, . . . , am) & satisfying 1/4-condition is exponentially 

generic. Hence the argument from Lemma 16.21 proves the required upper bound 



6.2 The Conjugacy Problems in free groups 

Now we turn to the conjugacy problems in free groups. Again, everywhere below 
G is a fixed group generated by a finite set X. 

It is easy to see that the CP and CSP in free groups arc decidable in at most 
quadratic time. It is quite tricky to show that CP and CSP arc decidable in 
free groups in linear time! This result is based on Knuth-Morris-Pratt substring 
searching algorithm [32j. Similarly, the Root Search Problem (listed below) is 
decidable in free groups in linear time. 

The Root Search Problem (RSP): Given a word w S F{X) find a shortest 
word u £ F{X) such that w = u" for some positive integer n. 

Notice, that RSP in free groups can be interpreted as a problem of finding 
a single generator of the centralizcr of a non-trivial clement. 

Theorem 6.5. The Simultaneous Conjugacy Problem (SCP) and Simultaneous 
Conjugacy Search Problem (SCSP) are in linear time reducible to CP, CSP, and 
RP in free groups. In particular, it is decidable in linear time. 

Proof. We briefly outline an algorithm that simultaneously solves the problems 
SCP and SCSP in free groups, i.e., given a finite system of conjugacy equations 



the algorithm decides whether or not this system has a solution in a free group 
F{X), and if so, it finds a solution. Using the decision algorithm for CP one can 
check whether or not there is an equation in ([5]) that does not have solutions 
in F. If so the whole system does not have solutions in F and we are done. 
Otherwise, using the algorithm to solve CSP in F one can find a particular 
solution di of every equation uf = in ([5]). In this case the set of all solutions 
of the equation uf — Vi is equal to the coset C{ui)di of the centralizcr C{ui). 
Observe, that using the decision algorithm for RSP one can find a generator 
(the root of Ui) of the centralizcr C{ui) in F. 

Consider now the first two equations in ([8]). The system 



for UMSP on T. 



□ 





Ui = Vi,U2 = V2 



(9) 
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has a solution in F(X) if and only if the intersection V = C{ui)di n C{u2)d2 is 
non-empty. In this case 

V = C{ui)di n C{u2)d2 - (C(ui) n C{u2)) d 

for some d £ F. 

If W2] = 1 then V, as the intersection of two cosets, is non-trivial if and 
only if the cosets coincide, i.e., = 1. This can be checked in linear 

time (since the word problem in F(X) is in linear time). Therefore, in linear 
time we cither check that the system hence the system ([5]), docs not have 
solutions at all, or we confirm that ^ is equivalent to one of the equations, so 
([5]) is equivalent to its own subsystem, where the first equation is removed. In 
the latter case induction finishes the proof. 

If [ui,U2] ^ 1 then C{ui) n C{u2) = 1, so either V = id or V ^ {d}, in both 
cases one can easily find all solutions of ([8]). Indeed, if = then ^ does not 
have solutions at all. li V = {d}, then d is the only potential solution of ([5]), 
and one can check whether or not d satisfies all other equations in ([5]) in linear 
time by the direct verification. 

Now the problem is to verify in linear time whether y = or not, which is 
equivalent to solving an equation 

di ^ uld2 (10) 

for integers m, k. Finding in linear time the cyclically reduced decompositions 
of ui and U2 one can rewrite the equation (jlOp into an equivalent one in the 
form: 

w^'^cw'^ = 6 (11) 

where WitW2 are cyclically reduced forms of ui,U2, and either w^^c or cwi (or 
both) are reduced as written, and b docs not begin with and does not end 
with wi- Again, in linear time one can find the maximal possible cancelation in 
w^'^c, and in cwi, and rewrite (jlip in the form: 

w^'^wl = b (12) 

where wi is a cyclic permutation of wi, and \b\ < \b\ + \wi\. Notice, that 
two cyclically reduced periodic words W2 , wi either commute or do not have a 
common subword of length exceeding |u'2| -I- \wi\. If they commute then the 
equation (|12p becomes a power equation, which is easy to solve. Otherwise, 
executing (in linear time) possible cancelation in the left-hand side of one 
arrives to an equation of the type 

w^^'ewl = b (13) 

where there is no cancelation at all. This can be easily solved for r and t. This 
proves the result. 

□ 
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As we have seen in the proof of Theorem 16.51 one of the main difficulties 
in solving SCSP in groups hes in computing the intersection of two finitely 
generated subgroups or their cosets. Notice, that finitely generated subgroups 
of F{X) are regular sets (which are accepted by their Stalhngs' automata). It 
is well known in the language theory that the intersection of two regular sets is 
again regular, and one can find an automaton accepting the intersection in at 
most quadratic time. This leads to the following corollary. 

Corollary 6.6. The SCSP* in free groups is decidable in at most quadratic 
time. 

Proof. Recall from the proof of Theorem 16. 51 that the algorithm solving a finite 
system of conjugacy equations in a free group either decides that there is no 
solution to the system, or produces a unique solution, or gives the whole solution 
set as a coset Cd of some centralizcr C. In the first case, the corresponding 
SCSP* has no solutions in a given finitely generated subgroup A; in the second 
case, given a unique solution w of the system one can construct the automaton 
Ta, that accepts A, and check whether w is in A or not (it requires nlog*n 
time); and in the third case, one needs to verify if Cd n A is empty or not - this 
can be done, as we have mentioned above, in at most quadratic time (as the 
intersection of two regular subsets). □ 

Observe from the proof above, that the most time consuming case in solving 
SCSP* in free groups occurs when all the elements iti, . . . , u„ in the system 
([8]) commute. The set of such inputs for SCSP* is, obviously, exponentially 
negligible. As we proved in Theorem 14.41 that LBA relative to It solves SCSP* 
in linear time. 

Since AAG is reducible in linear time to SCSP* (Lemma 13. 2|) we have the 
following results. 

Corollary 6.7. The following hold in an arbitrary free group F. 

1) The AAG algorithmic problem in F is decidable in at most quadratic 
time in the size of the input (the size of the public information in the 
AAG scheme). 

2) The AAG algorithmic problem in F is decidable in linear time on an 
exponentially generic set of inputs. 

6.3 The MSP and SCSP* problems in groups with "good" 
quotients 

In this section we discuss the generic complexity of the Membership Search 
Problem MSP and the Simultaneous Conjugacy Search Problem relative to a 
subgroup SCSP* in groups that have "good" factors in TBexp- 

Let G be a group generate by a finite set X, G/N is a quotient of G, and 
(j) : G G/N a canonical epimorphism. Let H = (ui, . . . ,Uk) be a finitely 
generated subgroup of G. To solve the membership search problem for H one can 
employ the following simple heuristic idea which we formulate as an algorithm. 
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Algorithm 6.8. (Heuristic solution to MSP) 

Input: A word w ~ w(A) and generators {ui, . . . , Uk} C F{X) of a subgroup 
H. 

Output: A representation W{ui, . . . , Uk) of w as an element of H or Failure. 
Computations: 

A. Compute the generators , ■ ■ ■ ,uf of H'^ in G/N . 

B. Compute w"^ , solve MSP for w'^ and if*^, and find a representation 

, . . . juf) oi w"^ as a product of the generators of itf , . . . , itf and 
their inverses. 

C. Check if W{ui, . . . , u^) is equal to w in G. If this is the case then output 
W . Otherwise output Failure. 

Observe that to run Algorithm 16.81 one needs to be able to solve MSP in the 
quotient G/N (Step B) and to check the result in the original group (Step C), 
i.e., to solve the Word Problem in G. If these conditions are satisfied Algorithm 
I6.8l is a partial deterministic correct algorithm, it gives only the correct answers. 
However, it is far from being obvious, even the conditions are satisfied, that this 
heuristic algorithm can be robust in any interesting class of groups. The next 
theorem, which is the main result of this section, states that Algorithm 16.81 is 
very robust for groups from J-Bexp with a few additional requirements. 

Theorem 6.9. (Reduction to a quotient) Let G be a group generated by a 
finite set X and with the Word Problem in a complexity class Gi(n). Suppose 
G/N is a quotient of G such that: 

1) G/N e TB,xp. 

2) The canonical epimorphism (j) : G G/N is computable within time 
C2{n). 

3) For every fc e N there exists an algorithm Ak in a complexity class Cz{n), 
which solves the Membership Search Problem in G/N for an exponentially 
generic set Mk C F{X)^ of descriptions of k-generated subgroups in G/N . 

Then for every k Algorithm 1 6. ^1 solves the Membership Search Problem on 
an exponentially generic set Tk C F{X)^ of descriptions of k-generated sub- 
groups in G. Furthermore, Algorithm 1 6. 8\ belongs to the complexity class 
Ci(n)+C2(n) + C3(n). 

Proof. We need to show that Algorithm [6?8] successfully halts on an exponen- 
tially generic set of tuples from F{X)^ . By the conditions of the theorem the 
set Sk of all fc-tuples from F{X)^ whose images in G/N freely generate free 
subgroups is exponentially generic, as well as, the set Mk of all tuples from 
F{X)^ where the algorithm Ak applies. Hence the intersection Tk = Sk H Mk 
is exponentially generic in F{X)''. We claim that Algorithm 16.81 applies to the 
subgroups with descriptions from Tk. Indeed, the algorithm Ak applies to sub- 
groups generated by tuples Y = (ui, . . . ,Uk) from Tk, so if w"^ G FT^ ~ (V^) 
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then Ak outputs a required representation = W{Y'^) in G/N. Notice, that 
H'^ is freely generated by Y'^' since Y £ Sk, therefore (p is injective on H. It 
follows that w = W{Y) in G, as required. This proves the theorem. □ 

Theorems 14.61 and 16.91 imply the following result. 

Corollary 6.10. Let G be as in Theorem 16.91 Then for every k,m > there 
exists an algorithm Ck.m that solves the SCSP* on an exponentially generic 
subset of the set of all inputs Ik,m for SCSP*. Furthermore, Ck,m belongs to the 
complexity class + Ci(n) + C2{n) + C^{n). 

Corollary 6.11. Let G be a group of pure braids PBm n > 3, or a non-abelian 
partially commutative group G{T). Then for every fc,TO > there exists an 
algorithm Ck,m that solves the SCSP* on an exponentially generic subset of the 
set of all inputs Ik,m for SCSP*. Furthermore, Ck.m belongs to the complexity 
class O(n^). 

Proof. Recall that the for any pure braid group or a non-abelian partially com- 
mutative group the Word problem can be solved by a quadratic time algorithm. 
Now the statement follows from Corollarv l6.10l and Corollaries 12.71 and 12.91 

□ 
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